Open Source Peer Review – Lessons and Recommendations for Closed Source

Although the effectiveness of software inspection (formal peer review) has a long and mostly supportive history, the thought of reviewing a large, unfamiliar software artifact over a period of weeks is something dreaded by both the author and the reviewers. The dislike of this cumbersome process is natural, but neither the formality nor the aversion are fundamental characteristics of peer review. The core idea of review is simply to get an expert peer to examine your work for problems that you are blind to. The actual process is much less important for finding defects than the expertise of the people involved [1]. While developers will acknowledge the value of inspection, most avoid it and adoption of traditional inspection practices remains relatively low [2], [3]. Surprisingly, peer review is a prevalent practice in Open Source Software (OSS) projects, which have minimalist processes and consist of intrinsically motivated developers. We examined over 100K peer reviews in case studies on the Apache httpd server, Subversion, Linux, FreeBSD, KDE, and Gnome and noted an “efficient fit” between the needs of OSS developers and the evolution and minimalist structure of the peer review processes they use [4]. On these OSS projects, changes are asynchronously broadcast (usually on a mailing list) to the development team, and reviewers self-select changes that they are invested in and competent to review. Changes that are uninteresting often remain unreviewed. To find changes in what can be an overwhelming broadcast of information, OSS developers rely on simple email filters, descriptive email subjects, and detailed change logs. These change logs represent the “the heart beat of the project.” Through them, developers maintain a conceptual understanding of the whole system and participate in the threaded email discussions and reviews for which they have the required expertise. This natural fit contrasts sharply with an enforced, but Figure 1. The spectrum of peer review techniques: From formal inspection to minimal-process OSS review. Tool supported, lightweight review provides a flexible, but traceable middle ground.